Objective:¶
TED talks are video recordings of influential talks given at and hosted by TED Conferences LLC. TED was founded in 1984, and has since built a reputation for spreading inspiring powerful ideas in fields ranging from tech to science to education. As video recordings of TED talks have garnered over 1 billion views to date, it is evident that TED represents a significant platform and opportunity for anyone with a powerful mission to raise awareness and attention to their work.
The goal of this project is to use various features associated with TED talks -- such as the topic, speaker, conference year, and length of the talk -- to predict the number of views a TED talk may receive. We also want to better understand what attributes are associated with popular TED Talks.¶
In [1]:
# Python ≥3.5 is required
import sys
assert sys.version_info >= (3, 5)
# Scikit-Learn ≥0.20 is required
import sklearn
assert sklearn.__version__ >= "0.20"
# Common imports
import numpy as np
import os
import pandas as pd
# To plot pretty figures
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
mpl.rc('axes', labelsize=14)
mpl.rc('xtick', labelsize=12)
mpl.rc('ytick', labelsize=12)
# Where to save the figures
PROJECT_ROOT_DIR = "."
CHAPTER_ID = "end_to_end_project"
IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, "images", CHAPTER_ID)
os.makedirs(IMAGES_PATH, exist_ok=True)
def save_fig(fig_id, tight_layout=True, fig_extension="png", resolution=300):
path = os.path.join(IMAGES_PATH, fig_id + "." + fig_extension)
print("Saving figure", fig_id)
if tight_layout:
plt.tight_layout()
plt.savefig(path, format=fig_extension, dpi=resolution)
# Ignore useless warnings (see SciPy issue #5998)
import warnings
warnings.filterwarnings(action="ignore", message="^internal gelsd")
pd.options.mode.chained_assignment = None # default='warn'
#display all columns
pd.set_option('display.max_columns', None)
In [2]:
import pandas as pd
import requests
import io
# Downloading the csv file from my GitHub account
url = "https://raw.githubusercontent.com/McGill-MMA-EnterpriseAnalytics/TED/main/data/ted_main.csv"
download = requests.get(url).content
# Reading the downloaded content and turning it into a pandas dataframe
# We will use "churn" instead of "df" for replication purposes
df = pd.read_csv(io.StringIO(download.decode('utf-8')))
# Printing out the first 5 rows of the dataframe
df.head()
Out[2]:
Data preprocessing¶
About the original dataset¶
- name: The official name of the TED Talk. Includes the title and the speaker.
- title: The title of the talk
- description: A blurb of what the talk is about.
- main_speaker: The first named speaker of the talk.
- speaker_occupation: The occupation of the main speaker.
- num_speaker: The number of speakers in the talk.
- duration: The duration of the talk in seconds.
- event: The TED/TEDx event where the talk took place.
- film_date: The Unix timestamp of the filming.
- published_date: The Unix timestamp for the publication of the talk on TED.com
- comments: The number of first level comments made on the talk.
- tags: The themes associated with the talk.
- languages: The number of languages in which the talk is available.
- ratings: A stringified dictionary of the various ratings given to the talk (inspiring, fascinating, jaw dropping, etc.)
- related_talks: A list of dictionaries of recommended talks to watch next.
- url: The URL of the talk.
- views: The number of views on the talk.
In [3]:
df.info()
In [4]:
# Reorganize the columns for a better visualization
df = df[['name', 'title', 'description', 'main_speaker', 'speaker_occupation', 'num_speaker', 'duration', 'event', 'film_date', 'published_date', 'comments', 'tags', 'languages', 'ratings', 'related_talks', 'url', 'views']]
df.head()
Out[4]:
In [5]:
# Some more information about the dataset
display(df.shape)
display(df.isnull().sum())
display(df.describe())
Initial Observations¶
- There are 2550 rows and 17 columns.
speaker_occupationcolumn have 6 missing values.
In [6]:
# working on df_copy for the rest of data exploration
df_copy = df.copy()
In [7]:
# Filtering out the TED talks
df_copy = df_copy[df_copy['event'].str.contains('TED', regex=False, case=False, na=False)]
df_copy.shape
Out[7]:
In [8]:
# Converting Duration to minutes
df_copy["duration"] = round(df_copy["duration"]/60,2)
In [9]:
df_copy.fillna('Unknown', inplace = True)
In [10]:
df_copy['languages'].describe()
Out[10]:
Observations with zero languages are musicals
In [11]:
df_copy[df_copy['languages'] == 0].head(3)
Out[11]:
In [12]:
#Indexing the rows
df_copy.reset_index(inplace=True)
In [13]:
from collections import defaultdict
rating_data = defaultdict(list)
In [14]:
import ast
rating_names = set()
for index, row in df_copy.iterrows():
rating = ast.literal_eval(row['ratings'])
for item in rating:
rating_names.add(item['name'])
In [15]:
rating_names
Out[15]:
In [16]:
#Extracting ratings
rating_data = defaultdict(list)
for index, row in df_copy.iterrows():
rating = ast.literal_eval(row['ratings'])
rating_data['ID'].append(row['index'])
names = set()
for item in rating:
rating_data[item['name']].append(item['count'])
names.add(item['name'])
rating_data = pd.DataFrame(rating_data)
rating_data.head()
Out[16]:
In [17]:
# Extracting tags
tags_data = defaultdict(list)
for index, row in df_copy.iterrows():
tags = ast.literal_eval(row['tags'])
for item in tags:
tags_data['ID'].append(row['index'])
tags_data['tags'].append(item)
tags_data = pd.DataFrame(tags_data)
In [18]:
tags_data[tags_data['ID']==1]
Out[18]:
In [19]:
# Extracting related talks
df_copy['related_views'] = 0
df_copy['related_duration'] = 0
for index, row in df_copy.iterrows():
rel = row['related_talks'].split(',')
ctr1 = 0
tot1 = 0
ctr2 = 0
tot2 = 0
for views in rel:
if 'viewed_count' in views:
view = views.split(':')
view[1] = view[1].replace("]", "")
view[1] = view[1].replace(" ", "")
view[1] = view[1].replace("}", "")
tot1+=int(view[1])
ctr1+=1
if 'duration' in views:
view = views.split(':')
view[1] = view[1].replace("]", "")
view[1] = view[1].replace(" ", "")
view[1] = view[1].replace("}", "")
tot2+=int(view[1])
ctr2+=1
df_copy['related_views'][index] = tot1/ctr1
df_copy['related_duration'][index] = tot2/ctr2
In [20]:
df_copy.head(3)
Out[20]:
In [21]:
df_copy['event_category'] = 'Other'
for i in range(len(df_copy)):
if df_copy['event'][i][0:5]=='TED20':
df_copy['event_category'][i] = 'TED2000s'
elif df_copy['event'][i][0:5]=='TED19':
df_copy['event_category'][i] = 'TED1900s'
elif df_copy['event'][i][0:4]=='TEDx':
df_copy['event_category'][i] = "TEDx"
elif df_copy['event'][i][0:7]=='TED@BCG':
df_copy['event_category'][i] = 'TED@BCG'
elif df_copy['event'][i][0:4]=='TED@':
df_copy['event_category'][i] = "TED@"
elif df_copy['event'][i][0:8]=='TEDSalon':
df_copy['event_category'][i] = "TEDSalon"
elif df_copy['event'][i][0:9]=='TEDGlobal':
df_copy['event_category'][i] = 'TEDGlobal'
elif df_copy['event'][i][0:8]=='TEDWomen':
df_copy['event_category'][i] = 'TEDWomen'
elif df_copy['event'][i][0:6]=='TEDMED':
df_copy['event_category'][i] = 'TEDMED'
elif df_copy['event'][i][0:3]=='TED':
df_copy['event_category'][i] = 'TEDOther'
In [22]:
# Convert timestamp into readable format
import datetime
df_copy['published_date'] = df_copy['published_date'].apply(lambda x: datetime.date.fromtimestamp(int(x)))
df_copy['day'] = df_copy['published_date'].apply(lambda x: x.weekday())
df_copy['month'] = df_copy['published_date'].apply(lambda x: x.month)
df_copy['year'] = df_copy['published_date'].apply(lambda x: x.year)
df_copy['film_date'] = df_copy['film_date'].apply(lambda x: datetime.date.fromtimestamp(int(x)))
df_copy['day_film'] = df_copy['film_date'].apply(lambda x: x.weekday())
df_copy['month_film'] = df_copy['film_date'].apply(lambda x: x.month)
df_copy['year_film'] = df_copy['film_date'].apply(lambda x: x.year)
In [23]:
to_cat = {"day": {0: "Monday", 1: "Tuesday", 2: "Wednesday", 3: "Thurday", 4: "Friday", 5: "Saturday",
6: "Sunday" },
"day_film": {0: "Monday", 1: "Tuesday", 2: "Wednesday", 3: "Thurday", 4: "Friday", 5: "Saturday",
6: "Sunday" }}
df_copy.replace(to_cat, inplace=True)
In [24]:
#create new attributes for length of the title, description, and for number of times the speaker spoke
df_copy['title_len'] = df_copy['title'].str.len()
df_copy['description_len'] = df_copy['description'].str.len()
df_copy['speaker_frequency'] = df_copy.groupby('main_speaker')['index'].transform('count')
df_copy['repeat_speaker'] = np.where((df_copy['speaker_frequency'] >1),1,0)
In [25]:
temp = tags_data.groupby(['tags']).count()
temp = temp.sort_values(by='ID',ascending=False)
In [26]:
temp.head(3)
Out[26]:
In [27]:
# Creating Tag Categories
df_copy['Technology/Science'] = 0
df_copy['Humanity'] = 0
df_copy['Global Issues'] = 0
df_copy['Art/Creativity'] = 0
df_copy['Business'] = 0
df_copy['Entertainment'] = 0
df_copy['Health'] = 0
df_copy['Communication'] = 0
df_copy['Education']=0
Tech = ['technology','future','comuters','science','invention','research']
Humanity = ['community','society','social change','humanity','culture']
Global_Issues = ['global issues','activism','politics','inequality','environment','climate change']
Art = ['design','art','innovation','creativity','brain']
Business = ['business','economics']
Entertainment = ['entertainment','media','sports']
Health = ['health','biollogy','medicine','health care','medical research']
Communication = ['communication','collaboration']
Education = ['children','education','teaching','parenting']
for i in range(len(tags_data)):
index = tags_data['ID'][i]
if tags_data['tags'][i] in Tech:
df_copy['Technology/Science'][index]=1
if tags_data['tags'][i] in Humanity:
df_copy['Humanity'][index]=1
if tags_data['tags'][i] in Global_Issues:
df_copy['Global Issues'][index]=1
if tags_data['tags'][i] in Art:
df_copy['Art/Creativity'][index]=1
if tags_data['tags'][i] in Business:
df_copy['Business'][index]=1
if tags_data['tags'][i] in Entertainment:
df_copy['Entertainment'][index]=1
if tags_data['tags'][i] in Health:
df_copy['Health'][index]=1
if tags_data['tags'][i] in Communication:
df_copy['Communication'][index]=1
if tags_data['tags'][i] in Education:
df_copy['Education'][index]=1
In [28]:
df_copy = df_copy.drop(['index','comments', 'event', 'film_date', 'main_speaker', 'name', 'published_date', 'ratings', 'url', 'description', 'title',
'related_talks', 'tags', 'title','speaker_occupation'], 1)
In [29]:
df_copy.head()
Out[29]:
In [30]:
import pandas_profiling
from pandas_profiling import ProfileReport
profile = ProfileReport(df_copy, title='Pandas Profiling Report', html={'style':{'full_width':True}})
profile
Out[30]:
II - Split Data¶
Due to the nature of the dataset, we decided to use a time based cross validation approach. The data itself is not time-series but still has a time dimension aspect (the date of the Ted Talk itself) which can be quite insightful for the current model we are building.
With the evolving popularity, environment, and subjects of TedTalks, it is best to use time-based splitting to build statistically robust models and follow up with time based cross validations to evaluate the performance of our final model.
The basis of the splits for the train and test data will be the year of release, with consideration with the amount of talks in each year to ensure a 80% and 20% split for training and test, respectively.
References: (which we can add at the end when we are done)
In [31]:
df_copy.year.value_counts().sort_index()
Out[31]:
First split: Training data for model building and cross validation and Test or "New data" -- data only used for final model evaluation¶
- Total number of observations: 2439
- Train (80%) = roughly 1952 observations
- Test (20%) = roughly 486 observations
Thus, based on the number of observations from each year, the first split will be
- Train : Ted Talks from 2006 until 2015 (2003 observations)
- Test: Ted Talks from 2016 and 2017 (436 observations)
Second split: Data for model building and cross validation¶
- Total number of observations of the train set : 2003 observations
- Train 2 set split (80%) : Ted talks from 2006 - 2013 (1548 observations)
- Test 2 set split (20%) : Ted talks from 2014 - 2015 (455 observations)
First split:¶
We perform the first split now, and do not touch the test data until the end. We perform our second split of the training dataset right before modeling.
In [32]:
#Train data that will further be split into train and test for model building and evaluation
train = df_copy[(df_copy["year"] >= 2006) & (df_copy["year"] <= 2015)]
# Test data ("new data") -- will be set aside and only be used for final model evaluation
test = df_copy[(df_copy["year"]) >= 2016]
In [33]:
train.columns
Out[33]:
In [34]:
# Plot some histograms on the training dataset
train.hist(bins=50, figsize=(20,15))
plt.show()
In [35]:
# Lets check for correlation
import seaborn as sns
from seaborn import pairplot
sns.set_style("whitegrid")
corr_continuous = train[['duration', 'languages','views','related_views','related_duration', 'title_len', 'description_len']]
plt.figure(figsize=(14,8))
sns.heatmap(corr_continuous.corr(), annot = True, cmap="Blues", linewidths=.1)
save_fig("correlation_matrix")
plt.show()
display(corr_continuous.corr())
- While duration have very weak correlation to views, number of speakers and different languages, suggesting longer talks don't seem to gather more attention from the audience.
In [36]:
print("Average duration = {}".format(round(train["duration"].mean(), 2)))
print("Median duration = {}".format(round(train["duration"].median(), 2)))
plt.figure(figsize=(10,5))
ax = sns.barplot(x="duration", y="event_category", data=train.sort_values('duration', ascending=False))
ax.set_title("Average TED Talk Duration by Category", pad=10, fontdict={'fontsize': 20})
plt.show()
Knowing that on an average most TED talks are usually about 18 mins long, average of 13.6 mins and median of 14.1 mins for the entore dataset seems a little lower.
TEDOther seems to be the most popular type of event. Infact most of the top duration talks aren't TED talks as seen from the event column. These talks were just hosted on ted.com website.
Having talks that over an hour are uncommon.
Views per category¶
In [37]:
print("Average views = {}".format(round(train["views"].mean(), 2)))
print("Standard Deviation (views) = {}".format(round(train["views"].std(), 2)))
print("Median views = {}".format(round(train["views"].median(), 2)))
plt.figure(figsize=(10,5))
ax = sns.barplot(x="views", y="event_category", data=train.sort_values('views', ascending=False))
ax.set_title("Average TED Talk Views by Category", pad=10, fontdict={'fontsize': 20})
plt.show()
- The most views came from TEDOther events, with almost 20 million views, which is huge when compared to average views of about 1.7 million, and a median of about 1.12 million.
- At second place is TEDx with more than 12.5 million views.
In [38]:
talk_years = train['year'].value_counts().reset_index()
talk_years.columns = ["year", "no_of_talks"]
plt.figure(figsize=(18,5))
ax = sns.barplot(x="year", y="no_of_talks", data=talk_years)
ax.set_title("TED Talks by Release Year", pad=10, fontdict={'fontsize': 20})
plt.show()
- Number of talks increased consistently from 2006 to 2012 and then stabilized.
Months¶
In [39]:
talk_years = train['month'].value_counts().reset_index()
talk_years.columns = ["month", "no_of_talks"]
plt.figure(figsize=(18,5))
ax = sns.barplot(x="month", y="no_of_talks", data=talk_years)
ax.set_title("TED Talks by Release Month", pad=10, fontdict={'fontsize': 20})
plt.show()
#1 = january ; 12 = December
Other Attributes vs. Target (Views)¶
The most popular TED talks:
- featured speakers that had spoken at a previous TED talk
- were on business topics
- were posted on Friday
- were around 20 minutes
- were translated into many languages
- had medium description lengths
In [40]:
plt.figure(figsize=(18,5))
ax = sns.barplot(x="repeat_speaker", y="views", data=train)
ax.set_title("TED Talks by Repeat Speaker", pad=10, fontdict={'fontsize': 20})
plt.show()
In [41]:
plt.figure(figsize=(18,5))
ax = sns.barplot(x="Business", y="views", data=train)
ax.set_title("TED Talks by Business or Not Business Category", pad=10, fontdict={'fontsize': 20})
plt.show()
In [42]:
plt.figure(figsize=(18,5))
ax = sns.barplot(x="day", y="views", data=train)
ax.set_title("TED Talk Views by Day of Week", pad=10, fontdict={'fontsize': 20})
plt.show()
In [43]:
sns.relplot(x ='duration', y = 'views', data = train)
sns.relplot(x ='languages', y = 'views', data = train)
sns.relplot(x ='description_len', y = 'views', data = train)
Out[43]:
Outlier Detection¶
We identified 51 outlier in this instance.
In [44]:
#use isolation forest to examine most extreme 2.5% of outliers from continuous variables from the analysis, which were visually identified during data exploration
continuous_vars = train[['duration', 'languages','views','related_views','related_duration']]
from sklearn.ensemble import IsolationForest
iforest = IsolationForest(n_estimators = 100,random_state=13, contamination=0.025)
#predict anomalies
pred = iforest.fit_predict(continuous_vars)
#anomaly score of each
iforest_scores = iforest.decision_function(continuous_vars)
#create a new variable to store the index number where anomaly =-1 in the anomalies vector
from numpy import where
anomaly_index = where(pred ==-1)
#extract values corresponding to the index of anomalies from the main df
anomaly_values = train.iloc[anomaly_index]
anomaly_values.head(3)
Out[44]:
Further Ted Talks exploration: Cluster Visualization¶
To better understand the dataset, we tried to identify some groups of TED talks that share similar characteristics.
In [45]:
#make a copy of this dataset for clustering at the end
train_clust = train.copy()
train_cluster = train_clust[['duration','views', 'languages']]
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
train_clust_std = scaler.fit_transform(train_cluster)
In [46]:
#Importing required modules
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
import numpy as np
#Transform the data
pca = PCA(2)
pca_cluster = pca.fit_transform(train_clust_std)
In [47]:
#Elbow Method
from sklearn.cluster import KMeans
withinss = []
for i in range(2,11):
kmeans = KMeans(n_clusters=i)
model_clustering = kmeans.fit(train_clust_std)
withinss.append(model_clustering.inertia_)
print('clusters: ', i, ', inertia: ', model_clustering.inertia_)
from matplotlib import pyplot
pyplot.plot([2,3,4,5,6,7,8,9,10],withinss)
#the optimal number of clusters is 4
Out[47]:
In [48]:
from sklearn.cluster import KMeans
kmeans = KMeans(n_clusters=4, random_state = 42)
model_clustering = kmeans.fit(pca_cluster)
labels = model_clustering.predict(pca_cluster)
from sklearn.metrics import silhouette_samples
from sklearn.metrics import silhouette_score
silhouette_score(pca_cluster,labels)
silhouette=silhouette_samples(pca_cluster,labels)
np.average(silhouette)
import matplotlib.pyplot as plt
#Getting the Centroids
centroids = kmeans.cluster_centers_
#Getting unique labels
u_labels = np.unique(labels)
#plotting the results:
for i in u_labels:
plt.scatter(pca_cluster[labels == i , 0] , pca_cluster[labels == i , 1] , label = i)
plt.scatter(centroids[:,0] , centroids[:,1] , s = 10,color="black")
plt.legend()
plt.show()
In [49]:
#number of observations in each cluster
from collections import Counter
Counter(labels)
Out[49]:
In [50]:
#group numbers into percentiles for easy visualization
cols = ['duration', 'views', 'languages']
new_cols = ['duration_p','views_p', 'languages_p']
for i in new_cols:
train_cluster[i] = 0
for k in range(len(cols)):
percentile = np.percentile(train_cluster[cols[k]], [25,50,75])
for i in range(len(train_cluster)):
if train_cluster[cols[k]][i] <= percentile[0]:
train_cluster[new_cols[k]][i] = '0-25'
elif train_cluster[cols[k]][i] <= percentile[1]:
train_cluster[new_cols[k]][i] = '26-50'
elif train_cluster[cols[k]][i] <= percentile[2]:
train_cluster[new_cols[k]][i] = '51-75'
else:
train_cluster[new_cols[k]][i] = '76-100'
In [51]:
#rename cluster labels
train_cluster.loc[:,'cluster_labels'] = labels
train_cluster.loc[:, 'event_category'] = train_clust['event_category']
train_cluster.loc[train_cluster['cluster_labels'] ==0, 'cluster_labels'] = 'Cluster 1'
train_cluster.loc[train_cluster['cluster_labels'] ==1, 'cluster_labels'] = 'Cluster 2'
train_cluster.loc[train_cluster['cluster_labels'] ==2, 'cluster_labels'] = 'Cluster 3'
train_cluster.loc[train_cluster['cluster_labels'] ==3, 'cluster_labels'] = 'Cluster 4'
train_cluster.loc[train_cluster['cluster_labels'] ==4, 'cluster_labels'] = 'Cluster 5'
In [52]:
train_cluster.head(3)
Out[52]:
In [53]:
import plotly.express as px
#CLUSTER DISTRIBUTION FOR DURATION
fig = px.histogram(train_cluster, x = 'cluster_labels', color= 'duration_p',
labels=dict(goal_usd1 = 'Percentile'),
category_orders={'duration_p': ['0-25', '26-50', '51-75', '75-100']}).update_xaxes(categoryorder='category ascending')
fig.update_layout(title_text='Clusters by Talk Duration (relative percentiles)', title_x=0.5)
fig.show()
In [54]:
import plotly.express as px
#CLUSTER DISTRIBUTION FOR LANGUAGES
fig = px.histogram(train_cluster, x = 'cluster_labels', color= 'languages_p',
labels=dict(goal_usd1 = 'Percentile'),
category_orders={'languages_p': ['0-25', '26-50', '51-75', '75-100']}).update_xaxes(categoryorder='category ascending')
fig.update_layout(title_text='Clusters by Num of Languages (relative percentiles)', title_x=0.5)
fig.show()
In [55]:
import plotly.express as px
#CLUSTER DISTRIBUTION FOR VIEWS
fig = px.histogram(train_cluster, x = 'cluster_labels', color= 'views_p',
labels=dict(goal_usd1 = 'Percentile'),
category_orders={'views_p': ['0-25', '26-50', '51-75', '75-100']}).update_xaxes(categoryorder='category ascending')
fig.update_layout(title_text='Clusters by Views (relative percentile)', title_x=0.5)
fig.show()
In [56]:
import plotly.express as px
#CLUSTER DISTRIBUTION FOR VIEWS
fig = px.histogram(train_cluster, x = 'cluster_labels', color= 'event_category',
labels=dict(goal_usd1 = 'Percentile'),
category_orders={'views_p': ['0-25', '26-50', '51-75', '75-100']}).update_xaxes(categoryorder='category ascending')
fig.update_layout(title_text='Clusters by Event Category (relative percentile)', title_x=0.5)
fig.show()
IV - Prepare data for modeling¶
Drop useless and invalid predictors¶
We first drop columns related to recommended videos as we assume that this will not be generated until the video has been posted online. We also assume that we cannot use languages as TED uses a volunteer based translation service where viewers can translate their favourite talks: https://www.ted.com/about/programs-initiatives/ted-translators. We assume this would not occur until after the video is posted also.
In [57]:
train.drop(['related_views','related_duration', 'languages'], axis = 1, inplace = True)
In [58]:
train
Out[58]:
Categorical Encoding¶
In [59]:
train_copy = pd.get_dummies(data = train, columns = ["event_category","day","month","day_film","month_film"])
In [60]:
train_crossval = train_copy.copy()
#this is the dataset that will be used for cross validation of the final models
In [61]:
# training data for the different models that will be tested
train2 = train[(train["year"] >= 2006) & (train["year"] <= 2013)]
# test data that will be use to evaluate all tested models
test2 = train[(train["year"]) >= 2014]
Second split : Data for model building and cross validation¶
In [62]:
y_train2 = train2['views']
X_train2 = train2.drop('views', axis = 1)
y_test2 = test2['views']
X_test2 = test2.drop('views', axis = 1)
In [63]:
y_test = test['views']
X_test = test.drop('views', axis = 1)
Preprocessing Pipeline¶
In [64]:
#Extracting Column names
X_train_copy = X_train2[["event_category","day","month","day_film","month_film"]]
X_train_copy_cat = pd.get_dummies(data = X_train_copy, columns = ["event_category","day","month","day_film","month_film"])
X_train_copy_num = X_train2.drop(["event_category","day","month","day_film","month_film"],axis=1)
num_cols=list(X_train_copy_num.columns)
cat_cols=list(X_train_copy_cat.columns)
col = cat_cols+num_cols
In [65]:
#Pipeline starts here
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
train_num = X_train2.drop(["event_category","day","month","day_film","month_film"], axis=1)
test_num = X_test.drop(["event_category","day","month","day_film","month_film"], axis=1)
num_pipeline = Pipeline([
('std_scaler', StandardScaler()),
])
train_num_tr = num_pipeline.fit_transform(train_num)
test_num_tr = num_pipeline.fit_transform(test_num)
from sklearn.compose import ColumnTransformer
num_attribs_train = list(X_train2.drop(["event_category","day","month","day_film","month_film"], axis=1))
num_attribs_test = list(X_test.drop(["event_category","day","month","day_film","month_film"], axis=1))
cat_attribs = ["event_category","day","month","day_film","month_film"]
full_pipeline = ColumnTransformer([
("cat", OneHotEncoder(), cat_attribs),
("num", num_pipeline, num_attribs_train),
])
train_prepared = full_pipeline.fit_transform(X_train2)
test_prepared2 = full_pipeline.fit_transform(X_test2)
test_prepared = full_pipeline.fit_transform(X_test)
#Convert array to dataframe for feature selection
#test and test2 datasets do not have the categories event_category_TED1900s, day_Saturday and day_Sunday.
#Hence, we add a dummy array to retain shape across the train and test datasets
a = np.zeros((455,))
b = np.zeros((436,))
test_preped2 = np.insert(test_prepared2, 0, a, axis=1)
test_preped2 = np.insert(test_preped2, 12, a, axis=1)
test_preped2 = np.insert(test_preped2, 13, a, axis=1)
test_preped = np.insert(test_prepared, 0, b, axis=1)
test_preped = np.insert(test_preped, 12, b, axis=1)
test_preped = np.insert(test_preped, 13, b, axis=1)
X_train2 = pd.DataFrame(train_prepared, columns=col)
X_test = pd.DataFrame(test_preped, columns=col)
X_test2 = pd.DataFrame(test_preped2, columns=col)
In [66]:
X_train2.head()
Out[66]:
In [67]:
X_test2.head()
Out[67]:
In [68]:
X_test.head()
Out[68]:
Feature selection¶
We now perform recursive feature elimination using random forest, and select top 35 features.
In [69]:
from sklearn.feature_selection import RFE
from sklearn.ensemble import RandomForestRegressor
rf = RandomForestRegressor(random_state=42)
model = RFE(rf, n_features_to_select=50)
fit_model = model.fit(X_train2, y_train2)
features = pd.DataFrame(list(zip(X_train2.columns,fit_model.ranking_)), columns = ['predictor','ranking'])
In [70]:
chosen_features = features.sort_values(by = 'ranking').head(35)['predictor'].tolist()
chosen_features
Out[70]:
In [71]:
X_train2 = X_train2[chosen_features]
X_test2 = X_test2[chosen_features]
In [72]:
X_test=X_test[chosen_features]
In [73]:
X_train2.shape
Out[73]:
In [74]:
X_test2.shape
Out[74]:
In [75]:
X_train2.info()
Scale Features¶
In [76]:
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train2)
X_train2 = pd.DataFrame(X_train_scaled)
X_test_scaled = scaler.fit_transform(X_test2)
X_test2 = pd.DataFrame(X_test_scaled)
In [77]:
#This pipeline doesn't work well with the above steps, not sure how to do feature selection with it
'''
train_num = X_train2.drop(["event_category","day","month","day_film","month_film"], axis=1)
train_cat = X_train2[["event_category","day","month","day_film","month_film"]]
test_num = X_test2.drop(["event_category","day","month","day_film","month_film"], axis=1)
test_cat = X_test2[["event_category","day","month","day_film","month_film"]]
from sklearn.preprocessing import OneHotEncoder
cat_encoder = OneHotEncoder()
cat_1hot_train = cat_encoder.fit_transform(train_cat)
cat_1hot_test = cat_encoder.fit_transform(test_cat)
cat_1hot_train.toarray()
cat_1hot_test.toarray()
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
num_pipeline = Pipeline([
('std_scaler', StandardScaler()),
])
train_num_tr = num_pipeline.fit_transform(train_num)
test_num_tr = num_pipeline.fit_transform(test_num)
train_num_tr
test_num_tr
from sklearn.compose import ColumnTransformer
num_attribs_train = list(X_train2.drop(["event_category","day","month","day_film","month_film"], axis=1))
num_attribs_test = list(X_test2.drop(["event_category","day","month","day_film","month_film"], axis=1))
cat_attribs = ["event_category","day","month","day_film","month_film"]
full_pipeline = ColumnTransformer([
("cat", OneHotEncoder(), cat_attribs),
("num", num_pipeline, num_attribs_train),
])
full_pipeline = ColumnTransformer([
("cat", OneHotEncoder(), cat_attribs),
("num", num_pipeline, num_attribs_test),
])
train_prepared = full_pipeline.fit_transform(X_train2)
test_prepared = full_pipeline.fit_transform(X_test2)
'''
Out[77]:
V - Select and train a model¶
In [78]:
#Import models' packages
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.linear_model import Lasso
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_squared_error, mean_absolute_error
#Regression Models
regression_models = [['RF regressor',RandomForestRegressor()],
['GBT',GradientBoostingRegressor()],
['LASSO', Lasso()],
['Ridge', Ridge()]]
#storing MSEs and MAEs for visualization
Model_MSEs = []
Model_MAEs = []
Models = ['Random Forest Regressor', 'Gradient Boosting Regressor', 'Lasso', 'Ridge']
for name, model in regression_models:
print('Model ', name, "performance: ")
model.fit(X_train2, y_train2.values.ravel())
y_test_pred = model.predict(X_test2)
MSE = mean_squared_error(y_test2, y_test_pred)
MAE = mean_absolute_error(y_test2, y_test_pred)
Model_MSEs.append(MSE)
Model_MAEs.append(MAE)
print("MSE: ", MSE)
print("MAE: ", MAE)
print("-" * 19)
In [79]:
y_test2.mean()
Out[79]:
In [80]:
len(Model_MSEs)
Out[80]:
Simple visualization of the initial results - MSE¶
In [81]:
d = {'Model' : Models, 'MSE': Model_MSEs}
MSE_vis = pd.DataFrame(d)
fig_dims = (15, 4)
fig, ax = plt.subplots(figsize=fig_dims)
ax = sns.barplot(x="Model", y="MSE", data=d, ax = ax)
ax.set_title("MSE of different tested models", pad=10, fontdict={'fontsize': 20})
ax.set_xlabel("Regression models",fontsize=20)
plt.show()
Simple visualization of the initial results - MAE¶
In [82]:
df_MAE = {'Model' : Models, 'MAE': Model_MAEs}
fig_dims = (15, 4)
fig, ax = plt.subplots(figsize=fig_dims)
ax = sns.barplot(x="Model", y="MAE", data=df_MAE, ax = ax)
ax.set_title("MAE of different tested models", pad=10, fontdict={'fontsize': 20})
ax.set_xlabel("Regression models",fontsize=20)
plt.show()
Testing more advanced regression models¶
- XGBoost
- Light GMB
- SVM
- Adabost
- KNN
I - XGBoost¶
In [83]:
import xgboost as xg
xgb_regressor = xg.XGBRegressor(objective ='reg:linear', n_estimators = 10, seed = 123)
# Fitting the model
xgb_regressor.fit(X_train2, y_train2.values.ravel())
# Predict the model
y_test_pred = xgb_regressor.predict(X_test2)
# MSE Computation
xgb_MSE = mean_squared_error(y_test2, y_test_pred)
print('The MSE of the XGBoost model is: ', xgb_MSE)
# MAE Computation
xgb_MAE = mean_absolute_error(y_test2, y_test_pred)
print('The MAE of the XGBoost model is: ', xgb_MAE)
# RMSE Computation
from sklearn.metrics import mean_squared_log_error
print('The RMSE of prediction for XGBoost is:', round(mean_squared_log_error(y_test2, y_test_pred) ** 0.5, 5))
In [84]:
# Parameter testing for base learner
# Train and test set are converted to DMatrix objects, as it is required by learning API.
train_dmatrix = xg.DMatrix(data = X_train2, label = y_train2)
test_dmatrix = xg.DMatrix(data = X_test2, label = y_test2)
# Parameter dictionary specifying base learner
param = {"booster":"gblinear", "objective":"reg:linear"}
xgb_r = xg.train(params = param, dtrain = train_dmatrix, num_boost_round = 10)
pred = xgb_r.predict(test_dmatrix)
# MSE Computation
xgb_MSE2 = mean_squared_error(y_test2, pred)
print('The MSE of the XGBoost model is: ', xgb_MSE2)
# MAE Computation
xgb_MAE2 = mean_absolute_error(y_test2, pred)
print('The MAE of the XGBoost model is: ', xgb_MAE2)
# RMSE Computation
#print('The RMSE of prediction is:', round(mean_squared_log_error(y_test2, pred) ** 0.5, 5))
## Storing the new MSEs and MAEs in original list
Model_MSEs.append(xgb_MSE2)
Model_MAEs.append(xgb_MAE2)
We can see an improvement in the MSE with the use of the parameter dictionnary
II - LightGBM¶
In [85]:
import lightgbm as lgb
hyper_params = {
'task': 'train',
'boosting_type': 'gbdt',
'objective': 'regression',
'metric': ['l2', 'auc'],
'learning_rate': 0.005,
'feature_fraction': 0.9,
'bagging_fraction': 0.7,
'bagging_freq': 10,
'verbose': 0,
"max_depth": 8,
"num_leaves": 128,
"max_bin": 512,
"num_iterations": 100000,
"n_estimators": 1000
}
gbm = lgb.LGBMRegressor(**hyper_params)
gbm.fit(X_train2, y_train2,
eval_set=[(X_test2, y_test2)],
eval_metric='l1',
early_stopping_rounds=1000)
Out[85]:
In [86]:
y_pred = gbm.predict(X_test2, num_iteration=gbm.best_iteration_)
# MSE Computation
lightgbm_MSE = mean_squared_error(y_test2, y_pred)
print('The MSE of LightGBM is: ', lightgbm_MSE)
# MAE Computation
lightgbm_MAE = mean_absolute_error(y_test2, y_pred)
print('The MAE of the LightGBM model is: ', lightgbm_MAE)
# RMSE Computation
print('The RMSE of prediction for LightGBM is:', round(mean_squared_log_error(y_test2, y_pred) ** 0.5, 5))
## Storing the new MSEs and MAEs in original list
Model_MSEs.append(lightgbm_MSE)
Model_MAEs.append(lightgbm_MAE)
III - SVM¶
In [87]:
from sklearn.svm import SVR
SVR_regressor = SVR(kernel = 'rbf')
SVR_regressor.fit(X_train2, y_train2)
#predict new results
y_pred_SVR = SVR_regressor.predict(X_test2)
# MSE Computation
SVM_MSE = mean_squared_error(y_test2, y_pred_SVR)
print('The MSE of SVM is: ', SVM_MSE)
# MAE Computation
SVM_MAE = mean_absolute_error(y_test2, y_pred_SVR)
print('The MAE of the SVM model is: ', SVM_MAE)
# RMSE Computation
print('The RMSE of prediction for SVM is:', round(mean_squared_log_error(y_test2, y_pred_SVR) ** 0.5, 5))
## Storing the new MSEs and MAEs in original list
Model_MSEs.append(SVM_MSE)
Model_MAEs.append(SVM_MAE)
IV - KNN Regressor¶
In [88]:
from sklearn import neighbors
MSE_val = [] #to store MSE values for different k
for K in range(0,20):
K = K+1
model_KNN = neighbors.KNeighborsRegressor(n_neighbors = K)
model_KNN.fit(X_train2, y_train2) #fit the model
pred_KNN = model_KNN.predict(X_test2) #make prediction on test set
MSE = mean_squared_error(y_test2, pred_KNN) #calculate MSE
MSE_val.append(MSE) #store MSE
print('MSE value for k= ' , K , 'is:', MSE)
In [89]:
#Plotting MSE values
curve = pd.DataFrame(MSE_val) #elbow curve
curve.plot()
Out[89]:
We can see that the lowest value of MSE is at K = 20
In [90]:
model_KNN = neighbors.KNeighborsRegressor(n_neighbors = 20)
model_KNN.fit(X_train2, y_train2) #fit the model
#predict new results
y_pred_KNN = model_KNN.predict(X_test2)
# MSE Computation
KNN_MSE = mean_squared_error(y_test2, y_pred_KNN)
print('The MSE of KNN is: ', KNN_MSE)
# MAE Computation
KNN_MAE = mean_absolute_error(y_test2, y_pred_KNN)
print('The MAE of the KNN model is: ', KNN_MAE)
# RMSE Computation
print('The RMSE of prediction for KNN is:', round(mean_squared_log_error(y_test2, y_pred_KNN) ** 0.5, 5))
## Storing the new MSEs and MAEs in original list
Model_MSEs.append(KNN_MSE)
Model_MAEs.append(KNN_MAE)
Visualization of all models tested¶
In [91]:
Model_MSEs
Out[91]:
In [92]:
Model_MAEs
Out[92]:
In [93]:
Models = ['Random Forest Regressor',
'Gradient Boosting Regressor',
'Lasso',
'Ridge',
'XGBoost',
'LightGBM',
'SVM',
'KNN']
d = {'Model' : Models, 'MSE': Model_MSEs}
MSE_vis = pd.DataFrame(d)
fig_dims = (20, 10)
fig, ax = plt.subplots(figsize=fig_dims)
ax = sns.barplot(x="Model", y="MSE", data=d, ax = ax)
ax.set_title("MSE of ALL models", pad=10, fontdict={'fontsize': 20})
ax.set_xlabel("Regression models",fontsize=20)
ax.set_xticklabels(ax.get_xticklabels(), rotation=30)
plt.show()
In [94]:
Models = ['Random Forest Regressor',
'Gradient Boosting Regressor',
'Lasso',
'Ridge',
'XGBoost',
'LightGBM',
'SVM',
'KNN']
d = {'Model' : Models, 'MSE': Model_MAEs}
MSE_vis = pd.DataFrame(d)
fig_dims = (20, 10)
fig, ax = plt.subplots(figsize=fig_dims)
ax = sns.barplot(x="Model", y="MSE", data=d, ax = ax)
ax.set_title("MAEs of ALL models", pad=10, fontdict={'fontsize': 20})
ax.set_xlabel("Regression models",fontsize=20)
ax.set_xticklabels(ax.get_xticklabels(), rotation=30)
plt.show()
In [95]:
all_models_performance = {'Model' : Models, 'MSE': Model_MSEs, 'MAEs': Model_MAEs}
performance = pd.DataFrame(all_models_performance)
performance
Out[95]:
In [96]:
#Rank the models based on MSE performance
performance.sort_values('MSE')
Out[96]:
Best Models¶
From the above results, we can see that the models with the lowest MSE are :
- LightGBM
- XGBoost
These are the models that will be fined tuned to find the final model that will be evaluated with our validation (initial test) dataset
Fine Tuning Top 2 Performing Models¶
Here we'll take the top 2 models and evaluate their performance.
TBC:
Cross validation will be also performed on both models.
LigthGBM: Fine tuning and Time-Based Cross validation¶
In [97]:
#parameters that we will tune
search_params = {'learning_rate': 0.001, #learning rate
'max_depth': 2000, #max depth of each trained tree -- has impact on model performance and training time
'num_leaves': 250, #important parameter -- controls complexity of model --> num leaves = 2^ (max_depth)
'feature_fraction': 0.8, #column sampling - % of features selected
'subsample': 0.2}
#parameters that should not change
fixed_params = {'objective': 'regression',
'metric': ['l2', 'auc'],
'boosting':'gbdt',
'num_boost_round':300,
'early_stopping_rounds':30}
#def train_evaluate(search_params):
train = lgb.Dataset(X_train2, label=y_train2)
valid_data = lgb.Dataset(X_test2, label=y_test2)
params = {'metric':fixed_params['metric'],
'objective':fixed_params['objective'],
**search_params}
gbm_TUNED = lgb.LGBMRegressor(**params)
gbm_TUNED.fit(X_train2, y_train2,
eval_set=[(X_test2, y_test2)],
eval_metric='l1',
early_stopping_rounds=10000)
num_iteration_GBM = gbm_TUNED.best_iteration_
#best_param = gbm_TUNED.best_params_
score = gbm_TUNED.best_score_
#return score, num_iteration_GBM
#train_evaluate(search_params)
#print("Best parameters found: ", gbm_TUNED.best_params_)
In [98]:
y_pred = gbm_TUNED.predict(X_test2, num_iteration=gbm_TUNED.best_iteration_)
# Comparing with original MSE, MAE and RMSE
print('----- LightGBM: Comparing with original MSE, MAE and RMSE -----')
print('---------------------------------------------------------------')
# MSE Computation
TUNED_lightgbm_MSE = mean_squared_error(y_test2, y_pred)
print('The MSE of tuned LightGBM is: ', TUNED_lightgbm_MSE)
print('MSE decreased: ', (1-(TUNED_lightgbm_MSE/lightgbm_MSE))*100, ' %')
print('---------------------------------------------------------------')
# MAE Computation
TUNED_lightgbm_MAE = mean_absolute_error(y_test2, y_pred)
print('The MAE of the tuned LightGBM model is: ', TUNED_lightgbm_MAE)
print('MAE decreased: ', (1-(TUNED_lightgbm_MAE/lightgbm_MAE))*100, ' %')
print('---------------------------------------------------------------')
# RMSE Computation
print('The RMSE of prediction for tuned LightGBM is:', round(mean_squared_log_error(y_test2, y_pred) ** 0.5, 5))
#print(lightgbm_MSE)
#print(TUNED_lightgbm_MSE)
XGBOOST : Fine tuning and Time-Based Cross validation¶
In [99]:
from sklearn.model_selection import RandomizedSearchCV
import xgboost as xgb
# Create the parameter grid: gbm_param_grid
gbm_param_grid = {
'n_estimators': [25],
'max_depth': range(2, 12)
}
# Instantiate the regressor: gbm
xgb_TUNED = xgb.XGBRegressor(n_estimators=10)
# Perform random search: randomized_mse
randomized_mse = RandomizedSearchCV(param_distributions=gbm_param_grid, estimator=xgb_TUNED,
scoring='neg_mean_squared_error', n_iter=5, cv=4,
verbose=1)
# Fit randomized_mse to the data
randomized_mse.fit(X_train2, y_train2, eval_set=[(X_test2, y_test2)])
# Print the best parameters and lowest RMSE
print("Best parameters found: ", randomized_mse.best_params_)
print("Lowest RMSE found: ", np.sqrt(np.abs(randomized_mse.best_score_)))
In [100]:
y_pred = randomized_mse.predict(X_test2)
"""
# Fitting the model
xgb_regressor.fit(X_train2, y_train2.values.ravel())
# Predict the model
y_test_pred = xgb_regressor.predict(X_test2)
"""
# Comparing with original MSE, MAE and RMSE
print('----- XGBoost: Comparing with original MSE, MAE and RMSE -----')
print('---------------------------------------------------------------')
# MSE Computation
xgb_MSE_tuned = mean_squared_error(y_test2, y_pred)
print('The MSE of tuned XGBoost is: ', xgb_MSE_tuned)
print('MSE decreased: ', (1-xgb_MSE_tuned/xgb_MSE)*100, ' %')
print('---------------------------------------------------------------')
# MAE Computation
xgb_MAE_tuned = mean_absolute_error(y_test2, y_pred)
print('The MAE of the tuned XGBoost model is: ', xgb_MAE_tuned)
print('MAE decreased: ', (1-xgb_MAE_tuned/xgb_MAE)*100, ' %')
print('---------------------------------------------------------------')
# RMSE Computation
print('The RMSE of prediction for tuned XGBoost is:', round(mean_squared_log_error(y_test2, y_pred) ** 0.5, 5))
Results - LightGBM vs XGBoost¶
In [101]:
final_models = ['LightGBM',
'XGBoost']
final_MSEs = [TUNED_lightgbm_MSE, xgb_MSE_tuned]
dff = {'Final_Model' : final_models, 'Final_MSE': final_MSEs}
MSE_vis = pd.DataFrame(dff)
fig_dims = (10, 5)
fig, ax = plt.subplots(figsize=fig_dims)
ax = sns.barplot(x="Final_Model", y="Final_MSE", data=dff, ax = ax)
ax.set_title("MSE of final models", pad=10, fontdict={'fontsize': 20})
ax.set_xlabel("Regression models",fontsize=20)
ax.set_xticklabels(ax.get_xticklabels(), rotation=30)
plt.show()
Time based cross validation (time series split)¶
Time based cross validation performed on the best performing model : LightGBM
In [102]:
import pandas as pd
import datetime
from datetime import datetime as dt
from dateutil.relativedelta import *
class TimeBasedCV(object):
'''
Parameters
----------
train_period: int
number of time units to include in each train set
default is 30
test_period: int
number of time units to include in each test set
default is 7
freq: string
frequency of input parameters. possible values are: days, months, years, weeks, hours, minutes, seconds
possible values designed to be used by dateutil.relativedelta class
deafault is days
'''
def __init__(self, train_period=30, test_period=7, freq='days'):
self.train_period = train_period
self.test_period = test_period
self.freq = freq
def split(self, data, validation_split_date=None, date_column='year', gap=0):
'''
Generate indices to split data into training and test set
Parameters
----------
data: pandas DataFrame
your data, contain one column for the record date
validation_split_date: datetime.date()
first date to perform the splitting on.
if not provided will set to be the minimum date in the data after the first training set
date_column: string, deafult='record_date'
date of each record
gap: int, default=0
for cases the test set does not come right after the train set,
*gap* days are left between train and test sets
Returns
-------
train_index ,test_index:
list of tuples (train index, test index) similar to sklearn model selection
'''
# check that date_column exist in the data:
try:
data[date_column]
except:
raise KeyError(date_column)
train_indices_list = []
test_indices_list = []
if validation_split_date==None:
validation_split_date = data[date_column].min().date() + eval('relativedelta('+self.freq+'=self.train_period)')
start_train = validation_split_date - eval('relativedelta('+self.freq+'=self.train_period)')
end_train = start_train + eval('relativedelta('+self.freq+'=self.train_period)')
start_test = end_train + eval('relativedelta('+self.freq+'=gap)')
end_test = start_test + eval('relativedelta('+self.freq+'=self.test_period)')
while end_test < data[date_column].max().date():
# train indices:
cur_train_indices = list(data[(data[date_column].dt.date>=start_train) &
(data[date_column].dt.date<end_train)].index)
# test indices:
cur_test_indices = list(data[(data[date_column].dt.date>=start_test) &
(data[date_column].dt.date<end_test)].index)
print("Train period:",start_train,"-" , end_train, ", Test period", start_test, "-", end_test,
"# train records", len(cur_train_indices), ", # test records", len(cur_test_indices))
train_indices_list.append(cur_train_indices)
test_indices_list.append(cur_test_indices)
# update dates:
start_train = start_train + eval('relativedelta('+self.freq+'=self.test_period)')
end_train = start_train + eval('relativedelta('+self.freq+'=self.train_period)')
start_test = end_train + eval('relativedelta('+self.freq+'=gap)')
end_test = start_test + eval('relativedelta('+self.freq+'=self.test_period)')
# mimic sklearn output
index_output = [(train,test) for train,test in zip(train_indices_list,test_indices_list)]
self.n_splits = len(index_output)
return index_output
def get_n_splits(self):
"""Returns the number of splitting iterations in the cross-validator
Returns
-------
n_splits : int
Returns the number of splitting iterations in the cross-validator.
"""
return self.n_splits
In [103]:
train_crossval
Out[103]:
In [104]:
#saving training data into CSV to parse with dates for cross validation
train_crossval.to_csv('train_data_ted_crossval.csv')
In [105]:
# How to use TimeBasedCV
data_for_modeling=pd.read_csv('train_data_ted_crossval.csv', parse_dates=['year'])
data_for_modeling.head()
Out[105]:
In [106]:
tscv = TimeBasedCV(train_period=10, #number of time units to include in each train set
test_period=2, #number of time units to include in each test set
freq='years') #frequency of input parameters
tscv
Out[106]:
In [107]:
for train_index, test_index in tscv.split(data_for_modeling,validation_split_date=datetime.date(2007,1,1),date_column='year'):
print(train_index)
print(test_index)
In [108]:
# get number of splits
tscv.get_n_splits()
Out[108]:
In [109]:
chosen_features
Out[109]:
In [110]:
if "year" not in chosen_features:
chosen_features += ["year"]
In [111]:
data_for_modeling
Out[111]:
In [112]:
#### Example- compute average test sets score: ####
X = data_for_modeling[chosen_features]
y = data_for_modeling['views']
In [113]:
X.head()
Out[113]:
In [114]:
from sklearn.linear_model import LinearRegression
import numpy as np
scores = []
for train_index, test_index in tscv.split(X, validation_split_date=datetime.date(2007,1,1)):
data_train = X.loc[train_index].drop('year', axis=1)
target_train = y.loc[train_index]
data_test = X.loc[test_index].drop('year', axis=1)
target_test = y.loc[test_index]
clf = LinearRegression()
clf.fit(data_train,target_train)
preds = clf.predict(data_test)
# accuracy for the current fold only
r2score = clf.score(data_test,target_test)
scores.append(r2score)
In [115]:
#### Example- RandomizedSearchCV ####
from sklearn.model_selection import RandomizedSearchCV
from lightgbm import LGBMRegressor
from random import randint, uniform
tscv = TimeBasedCV(train_period=7, test_period=3,freq='years')
index_output = tscv.split(data_for_modeling, validation_split_date=datetime.date(2007,1,1))
lgbm = LGBMRegressor()
lgbmPd = {" max_depth": [-1,2]
}
model = RandomizedSearchCV(
estimator = lgbm,
param_distributions = lgbmPd,
n_iter = 10,
n_jobs = -1,
iid = True,
cv = index_output,
verbose=5,
pre_dispatch='2*n_jobs',
random_state = None,
return_train_score = True)
model.fit(X.drop('year', axis=1),y)
model.cv_results_
Out[115]:
Feature Importance¶
We will perform Feature importance on the LightGBM model.
In [116]:
# LightGBM for feature importance on a regression problem
from lightgbm import LGBMRegressor
from matplotlib import pyplot
# define the model
model = LGBMRegressor()
# fit the model
model.fit(X_train2, y_train2)
# get importance
importance = model.feature_importances_
# summarize feature importance
for i,v in enumerate(importance):
print('Feature: %0d, Score: %.5f' % (i,v))
# plot feature importance
pyplot.bar([x for x in range(len(importance))], importance)
pyplot.show()
features = pd.DataFrame(list(zip(chosen_features,importance)), columns = ['predictor','importance'])
print(features)
Final Model Performance¶
In [121]:
final_predictions = gbm_TUNED.predict(X_test)
print('---------------------------------------------------------------')
# RMSE Computation
print('The RMSE of prediction for tuned LightGBM is:', round(mean_squared_log_error(y_test, final_predictions) ** 0.5, 5))